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user's  manual  for  the  CATS  program.  CATS  is  an  acronym  for  computer-based 
instruction  authoring  tools  system,  a  government-owned  set  of  software  tools  for 
constructing  and  presenting  interactive  courseware  on  a  personal  computer.  The  three 
supplemental  reports  concern  lesson  presentation  (NPRDC-TN -90-28),  lesson 
maintenance  (NPRDC-TN-90-30),  and  a  guide  for  students  (NPRDC-TN -90-29). 

The  ultimate  goal  of  the  GTET  project  is  to  provide  guidelines  for  the  training 
manager  responsible  for  converting  residential  courses  into  transportable  ones.  In  this 
report,  the  authors  document  the  effectiveness  of  prototype  transportable  lessons  by 
looking  at  student  performance.  The  most  important  finding  within  the  constraints  of  the 
study  is  that  transportable  courseware  is  at  least  as  effective  as  residential  courseware  in 
teaching  students  nontechnical  material  and  is  a  far  more  cost-effective  approach. 
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SUMMARY 


PROBLEM 

It  is  becoming  increasingly  important  to  the  military  to  develop  alternatives  to 
residential  classroom  instruction,  which  is  expensive,  usually  unstandardized,  group- 
oriented,  and  limited  to  small  numbers  of  students.  Transportable  courseware  is  such  an 
alternative. 


RESEARCH  FOCUS 

Many  people  throughout  the  Department  of  Defense  need  basic  information  about 
acquisition  management,  but  the  demand  for  this  kind  of  training  is  greater  than  the 
residential  schools  can  satisfy.  To  determine  if  residential  courses  in  acquisition 
management  could  be  readily  converted  into  transportable  ones,  the  program  sponsors 
selected  the  Defense  System  Management  College’s  (DSMC)  residential  Program 
Management  Course  (PMC)  to  serve  as  the  focus  of  a  research  effort. 

The  objective  of  this  effort  was  to  develop  transportable  lessons  that  were  as 
educationally  effective  as  residential  lessons  and  that  could  serve  as  an  adequate 
substitute  for  classroom  instruction  under  the  premise  that  transportable  instruction  is 
more  economical  to  use  because  it  can  reach  far  greater  numbers  of  students. 


APPROACH 

Transportable  lessons  were  developed  for  one  of  the  seven  modules  that  introduce 
students  at  the  DSMC  to  the  PMC.  The  transportable  lessons  for  this  module,  called 
Tools  for  Program  Management  (TPM),  were  prepared  in  both  print  and  computer 
versions. 

The  transportable  lessons  were  created  using  a  systems  approach  (the  SEID 
process  or  Systems  Engineering  of  Instructional  Development  process).  Lessons  were 
written  from  a  set  of  lesson  specifications  and  objectives,  and  test  items  on  the  final  TPM 
test  reflected  those  objectives. 

To  determine  whether  these  transportable  lessons  could  serve  as  an  adequate 
substitute  for  the  lessons  in  the  residential  course,  31  students  from  DSMC  were  selected 
to  be  members  of  a  control  group;  they  received  conventional  residential  instruction  on 
the  TPM  module  as  part  of  the  PMC  introduction.  Classroom  presentation  of  the  TPM 
material  was  interspersed  over  3  of  the  6  weeks  of  the  course.  Most  of  the  instruction 
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was  given  in  a  lecture/discussion  style.  Fifteen  students  from  the  Air  Force  Institute  of 
Technology  and  14  from  the  Systems  Acquisition  School  served  as  the  experimental 
group,  learning  about  TPM  through  the  newly  developed  transportable  lessons.  Half  of 
the  students  from  each  experimental  site  received  the  print  version  of  the  TPM  lessons 
and  half  received  the  computer  version. 

Both  groups  were  given  the  same  final  test  made  up  of  55  multiple -choice  items. 
The  students  in  the  control  group  were  tested  together  on  the  TPM  module  after  the  fifth 
week  of  the  6-week  PMC.  The  students  in  the  experimental  group  were  tested 
individually  as  each  finished  the  lessons. 


RESULTS  AND  DISCUSSION 

The  key  finding  was  that  students  who  studied  with  the  transportable  lessons 
performed  at  least  as  well  as  students  who  took  the  residential  classroom  lessons. 
Students  who  took  the  transportable  lessons  actually  scored  higher  than  classroom 
students.  But,  since  the  teaching  methods  used  by  the  instructors  may  have  differed 
greatly  within  the  control  group  setting  and  because  control  group  learning  of  the  TPM 
module  took  place  within  a  larger  context  (i.e.,  as  part  of  the  PMC  introduction),  these 
score  differences  cannot  be  attributed  to  the  transportable  media  alone. 

There  was  no  significant  difference  in  scores  between  those  students  in  the 
experimental  group  who  received  the  print  version  and  those  who  received  the  computer- 
based  version. 


CONCLUSIONS 

Within  the  constraints  of  this  study,  the  authors  found  that  transportable 
instruction  can  be  as  effective  as  residential  classroom  instruction  in  presenting 
nontechnical  information  to  acquisition  management  students. 


RECOMMENDATIONS 

(1)  A  systematic  front-end  training  analysis  can  help  determine  if  transportable 
lessons  are  an  appropriate  medium  for  a  given  subject  matter  area. 

(2)  Since  no  special  educational  benefits  can  be  attributed  to  any  one  presentation 
medium,  the  most  cost-effective  medium  should  be  chosen  to  present  lessons. 
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INTRODUCTION 


Problem 

The  military  services,  including  the  reserve  community,  need  to  provide  quality  training 
in  both  technical  and  nontechnical  areas  to  large  and  varied  populations  of  students  at  widely 
distant  locations.  Of  particular  concern  to  the  military  services  is  the  training  developed  for 
those  responsible  for  the  acquisition  of  weapons  and  other  supplies,  with  the  goal  that  all 
materiel  received  performs  well  and  is  priced  reasonably.  Congress  has  also  advocated 
strengthening  acquisition  education  and  has  recommended  improving  defense  management,  in 
general,  and  the  acquisition  system,  in  particular  (Packard  Commission,  June  1986).  A  recent 
Assistant  Secretary  of  Defense  stated  that  a  necessary  near-term  action  is  the  development  of 
better  qualified  procurement  personnel.  He  noted  too  the  need  to  "promote  wider  use  of  non- 
traditional  training  methods."1 

Residential  instruction,  however,  does  not  provide  all  the  answers.  It  has  many 
limitations,  among  them:  (1)  lack  of  uniformity  in  lesson  content  (sometimes  due  to  lack  of 
clear  training  objectives,  sometimes  due  to  varying  instructor  experience,  interest  and 
capabilities);  (2)  lack  of  a  systematic  process  by  which  to  update  and  revise  course  materials  so 
that  they  are  the  same  for  all  instructors  in  all  classrooms;  (3)  nonadaptability  to  an  individual's 
learning  needs;  (4)  inaccessibility  to  all  students  who  need  or  desire  the  training;  and  (5)  rigid 
scheduling  that  requires  students  to  delay  training  until  it  is  offered  in  a  classroom.  Finally, 
residential  training  is  very  expensive. 

Transportable  education  addresses  many  of  the  limitations  of  residential  training  and  has 
proven  to  be  more  cost-efficient.  It  does  not  require  sending  personnel  away  from  the  work  site 
for  training.  Lessons  are  standardized,  which  means  that  everyone  taking  the  course  will  receive 
the  same  message.  Ultimately,  many  more  people  can  be  trained  for  fewer  dollars  (Locatis, 
1987). 


Transportable  education  permits  more  timely  training;  courses  can  be  available  for 
presentation  whenever  the  student  or  supervisor  determines  they  are  needed.  These  courses  can 
also  be  used  as  cost-efficient  refreshers  for  people  who  need  to  be  brought  up  to  date  on  a 
particular  topic.  If  carefully  designed,  transportable  lessons  can  provide  other  time  efficiencies 
because  they  can  be  structured  to  take  advantage  of  each  student's  skill  and  knowledge  and 
permit  students  to  choose  the  amount  and  kind  of  instruction  they  wish  to  receive. 

Although  the  need  for  transportable  education  in  professional  management  programs  is 
well  documented,  there  are  few  guidelines  on  how  to  develop  computer-based  lessons  in 
nontechnical  areas. 


William  Taft,  19  August  1985,  in  a  Memorandum  to  the  Service  Secretaries  and  Assistant  Secretaries  of  Defense. 
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Research  Focus 


Earlier  research  funded  by  the  Naval  Air  Systems  Command  (NAVAIR)  explored  the 
feasibility  of  transporting  residential  acquisition  management  instruction.  To  determine  if 
residential  courses  in  acquisition  management  could  be  converted  into  transportable  ones, 
NAVAIR,  the  Defense  Systems  Management  College  (DSMC),  and  the  Navy  Personnel 
Research  and  Development  Center  (NAVPERSRANDCEN)  selected  the  residential  Program 
Management  Course  (PMC)  at  the  Defense  Systems  Management  College  (DSMC)  to  serve  as 
the  focus  of  this  research  effort.2 

Much  of  the  material  presented  in  the  PMC  is  similar  to  material  in  courses  provided  at 
other  locations,  such  as  those  provided  by  the  Air  Force  Institute  of  Technology  (AFIT)  and  the 
Systems  Acquisition  School  (SAS)  of  the  Air  Force  Systems  Command  (AFSC).  Since 
transportable  training  can  be  broken  up  into  modules  and  used  in  a  wide  variety  of  settings,  it 
was  reasoned  that  portions  of  the  transportable  materials  could  substitute  for  some  lessons  from 
these  other  courses,  making  the  lessons  useful  to  an  even  wider  population. 

Personnel  from  the  three  agencies  noted  above  proposed  to  provide  students  with  a 
significant  portion  (approximately  7  days)  of  the  PMC  material  in  a  transportable  instructional 
package.  Two  versions  of  the  transportable  material  would  be  created:  a  print  version  and  a 
computer-based  version. 


Hypotheses 

Course  developers  predicted  that  transportable  instructional  materials  would  provide 
nonresidential  education  that  was  at  least  equal  in  value  to  that  of  its  parallel  residential  course  in 
terms  of  student  performance.  The  objective  of  the  project  was  to  create  transportable 
courseware  that  could  substitute,  not  surpass,  classroom  training.  Course  developers  also 
hypothesized  that  there  would  be  no  differences  between  the  transportable  print  and  computer 
courseware,  since  the  computer  lessons  paralleled  the  print  lessons  in  every  way-content, 
organization,  and  format.  For  the  print  version,  one  turns  a  page;  for  the  computer  version,  one 
presses  a  spacebar  to  move  from  screen  to  screen.  Meta-analyses  by  both  Clark  (1985)  and 
Kulik  (Kulik,  Kulik,  &  Cohen,  1980)  indicate  that  most  performance  differences  that  have  been 
found  between  classroom  and  computer-based  versions  of  instruction  are  due  to  variables  such 
as  instructional  methods  rather  than  to  instructional  delivery  methods.  Since  the  print  and 
computer  lessons  were  basically  the  same,  no  differences  were  anticipated  between  these  two 
versions. 

2  Another  product  of  this  present  effort  is  a  set  of  Guidelines  for  Transportable  Education  and  Training  (GTET), 
guidelines  that  project  managers  can  use  to  help  develop  and  implement  transportable  instructional  packages.  These 
guidelines  are  in  the  process  of  being  developed  and  are  based,  in  part,  on  the  lessons  learned  from  this  effort 
(Tarker,  Rybowiak,  Flaningam,  &  Hulton,  1990). 


APPROACH 


Materials 

Analysis  and  Design  of  Materials 

A  training  needs  analysis  was  conducted  that  resulted  in  three  products:  (a)  a  topical 
outline  that  provided  course  objectives  for  several  lessons;  (b)  lesson  specifications  that  evolved 
from  the  topical  outline  and  that  were  used  to  guide  the  design  and  development  of  the  prototype 
course  materials;  and  (c)  tradeoff  analyses  that  determined  appropriate  training  media,  including 
hardware  and  software  for  both  courseware  authoring  and  development  and  for  lesson 
presentation. 

Courseware  Development 

Systems  Engineering  of  Instructional  Development  (SEID)  technology  and  computer- 
based  authoring  systems  were  used  as  tools  in  the  design,  development,  and  presentation  of 
transportable  education  lessons  (Instructional  Science  and  Development,  Inc.,  1987).3  For  this 
project,  a  transportable  print  version  and  a  computer  version  were  used  to  present  the  courseware 
to  students. 

Development  of  the  courseware  was  based  on  lesson  specifications.  The  lessons  were 
produced  iteratively  to  maintain  quality  and  cost  controls.  They  were  produced  by  teams  of 
contract  and  government  personnel,  with  subject  matter  expert  review  and  input  by  DSMC 
personnel.  All  activities  were  closely  monitored  to  ensure  the  correct  application  of  systematic 
instructional  technology  methods. 

The  courseware  was  designed  to  help  students  spend  their  time  more  efficiently  (e.g.,  to 
spend  more  time  on  areas  that  are  causing  them  the  greatest  difficulty).  The  materials  were 
designed  so  that  students  with  higher  aptitudes  or  greater  experience  were  able  to  proceed 


3SEID  evolved  from  the  Instructional  Systems  Development  (ISD)  approach  that  was  formalized  in  the  late  1960s 
and  early  1970s.  SEID  is  similar  to  ISD  in  the  use  of  a  systematic  approach  to  all  phases  of  instructional 
development,  ranging  from  analysis,  design,  development,  and  implementation  through  evaluation  and  management. 
The  SEID  differs  in  the  sense  that  it  was  designed  for  nontechnical  training,  such  as  acquisition  management 
education,  while  ISD  was  designed  for  use  with  highly  technical  courses,  such  as  those  that  teach  how  to  operate  or 
repair  machinery. 
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through  the  course  material  quickly,  while  those  with  lower  aptitudes  or  less  experience  were 
able  to  receive  additional  assistance  as  needed. 

A  formative  evaluation  of  the  materials  was  conducted.  Both  students  and  teachers  who 
used  the  materials  provided  the  feedback.  Some  of  the  lessons  were  extensively  revised  as  a 
result  of  this  evaluation,  and  many  other  format  and  content  recommendations  were  incorporated 
into  the  final  materials. 

Further  details  on  courseware  design  and  development  are  provided  in  a  related  report 
entitled  Lessons  Learned  in  Converting  Residential  Courseware  to  Transportable  Courseware 
(Tarker  et  al.,  1990). 

Transportable  Lessons 

The  transportable  lessons  consisted  of  seven  lessons  from  one  module  of  the  Introductory 
PMC  (PMC  (I))  course--Tools  for  Program  Management  (TPM).  In  the  classroom  setting,  the 
content  of  these  lessons  was  presented  over  a  6-week  period.  (See  Appendix  A  for  a  description 
of  the  PMC(I))  modules  and  the  TPM  module  that  was  converted  into  transportable  lessons.) 

Two  versions  of  each  transportable  lesson  were  developed:  a  self-paced  print  version  and 
a  self-paced  computer  version.  Both  versions  were  designed  to  be  used  by  individuals  at  or  near 
their  workplaces. 

Each  student  received  either  a  print  version  or  a  computer  version  of  each  of  the  seven 
lessons.  Both  modes  were  designed  to  require  minimal  interaction  with  an  instructor,  needing 
only  a  facilitator  who  would  be  available  to  answer  questions.  The  facilitator  was  either  present 
during  all  lessons  or  was  available  by  phone.  Students  planned  their  own  time,  working  on  the 
lessons  as  time  permitted.  Both  versions  contained  the  same  content  material  and  only  differed 
in  presentation  method.  All  lessons  included  a  pre-quiz  and  post-quiz,  an  introduction  and  a  list 
of  lesson  objectives,  content  information,  a  summary,  and  practice  questions.  Some  lessons  also 
contained  supplementary  information  and  examples. 

Each  print-version  lesson  consisted  of  a  printed  booklet  (8  1/2"  by  11")  that  contained  all 
of  the  instructional  materials.  Each  computer  version  lesson  consisted  of  a  student  disk  and  a  set 
of  5-1/4"  floppy  disks.  Each  student  received  his/her  own  floppy  disk  that  stored  the  student's 
test  results  and  tracked  the  student’s  completion  of  lesson  sections.  All  of  the  students  using  the 
computer  version  received  a  printed  user's  guide  for  each  lesson  that  contained:  (1)  instructions 
for  installing  and  using  the  computer  lessons,  and  (2)  a  variety  of  appendices  with  printed  pages 
of  some  of  the  lesson  content.  All  material  in  the  computer  lesson  appendices  was  also  part  of 
the  print  lessons. 

The  microcomputers  used  to  present  the  computer  courseware  were  off-the-shelf  systems 
(Zenith  248s)  that  already  existed  at  the  students'  work  sites  or  at  a  training  location  nearby. 
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Evaluation  Materials 


All  students  who  used  the  transportable  lessons  also  received  a  feedback  form  packet  that 
asked  questions  about  the  lessons.  A  list  of  the  feedback  questions,  including  questions 
soliciting  demographic  information,  is  included  in  Appendix  B. 

Demographic  information  was  gathered  from  each  student.  Questions  covered  previous 
education,  military  rank  or  civilian  grade,  and  work-related  experience  in  the  systems 
acquisition  field.  After  the  students  finished  all  of  the  lessons,  they  were  asked  about  their 
previous  experience  with  the  content  presented  in  each  of  the  lessons.  Attitudes  toward  the 
lessons  were  assessed  through  the  use  of  four  basic  ranking  questions  about  each  lesson,  with 
space  provided  for  written  comments.  Students  were  also  asked  to  record  the  number  of  minutes 
it  took  them  to  complete  each  lesson. 

Coordinator  Materials 

A  coordinator  at  each  testing  site  was  also  provided  with:  (1)  a  description  of  the  purpose 
of  the  evaluation,  (2)  directions  for  protecting  the  identity  of  the  students,  (3)  a  list  of  materials 
for  each  student,  (4)  a  description  of  the  students'  need  of  a  content-knowledgeable  facilitator 
(this  was  usually  the  coordinator)  who  could  answer  questions  about  subject  matter,  and  (5) 
directions  for  how  and  when  to  return  the  materials.  A  copy  of  the  instructions  to  coordinators 
is  provided  in  Appendix  C.  Coordinators  were  also  provided  with  the  final  examination  and 
instructed  to  monitor  its  use. 


Procedures 

All  materials  were  mailed  to  the  coordinators  at  the  training/evaluation  sites.  The  final 
results  and  feedback  were  returned  as  the  materials  were  completed,  about  4  months  later.  The 
coordinators  were  responsible  for  identifying  and  eliciting  the  cooperation  of  representative 
volunteer  students  to  receive  the  transportable  instruction.  Coordinators  were  also  tasked  with: 
(1)  distributing  and  collecting  the  materials,  (2)  explaining  to  the  students  the  importance  of 
doing  the  lessons  and  providing  feedback,  (3)  tracking  the  materials  with  identification  numbers 
to  preserve  participant  anonymity,  and  (4)  determining  how  to  present  the  computer  lessons  (see 
below). 


Each  training  site  set  up  its  own  delivery  and  installation  system  for  the  computer 
lessons.  At  SAS,  the  computer  lessons  were  installed  on  computers  by  the  coordinator,  not  the 
students.  At  AFIT,  the  coordinator  distributed  the  computer  disks  to  the  students,  who  then 
installed  them  on  their  own  computers  and  completed  the  lessons  at  their  own  work  sites. 
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The  instructions  recommended  that  students  devote  no  more  than  2  hours  at  a  time  to  the 
coursework,  for  both  print  and  for  computer  versions.  Of  course,  it  was  left  up  to  the  individual 
students  whether  or  not  to  follow  these  recommendations. 

Half  of  the  students  at  each  experimental  site  used  the  computer  version  and  half  used  the 
print  version. 


Subjects 

Personnel  from  three  sites  participated  in  the  evaluation.  DSMC  provided  31  students 
from  one  of  its  regional  centers  to  receive  conventional  residential  instruction  and  to  serve  as  the 
control  group.  AFIT  provided  15  students  and  SAS  provided  14  students  to  receive  the 
transportable  instruction. 

The  students  who  received  the  residential  classroom  instruction  were  students  at  DSMC. 
They  were  given  the  introductory  portion  of  PMC  over  a  6-week  period  at  DSMC's  regional 
center  in  Huntsville,  Alabama. 

The  two  Air  Force  sites  were  selected  because  their  students  represent  the  ultimate  target 
population  in  terms  of  education  and  experience.  These  students  were  potential  candidates  for 
the  PMC  in  that  they  met  DSMC's  experience  and  educational  requirements.  Students  at  both 
sites  were  volunteers.  One  site  offered  course  credit  for  participation. 

Demographic  information  on  all  subjects  is  presented  in  Table  1.  The  experimental 
groups  were  mainly  Air  Force  students  (with  AFIT  having  two  non-Air  Force  students).  The 
control  group  (Huntsville)  was  more  heterogeneous;  the  largest  single  group  represented  was 
from  the  Army  (54.8%).  The  majority  of  the  Huntsville  students  (51.6%)  and  AFIT  students 
(53.3%)  were  civilians  at  the  GS-11  level  and  above.  The  SAS  students  formed  a  more 
homogeneous  group  in  terms  of  service  and  grade,  with  all  students  being  Air  Force  military 
personnel,  primarily  with  an  "01  to  03"  rank  (85.7%).  The  remainder  of  the  SAS  students  (2) 
were  ranked  "04  and  up"  (14.3%). 

No  student  at  either  of  the  experimental  sites  had  previously  attended  the  introductory 
PMC  course.  Six  (42.9%)  of  the  SAS  students  were  instructors;  none  of  the  students  from 
Huntsville  or  AFIT  functioned  as  teachers. 

Data  on  degree,  years  of  acquisition  experience,  years  of  program  office  experience,  and 
years  of  military/federal  service  were  only  collected  for  the  two  experimental  sites.  All  the 
students  at  both  experimental  sites  had  college  degrees.  AFIT  students  reported  either  a 
bachelor's  or  a  master's  degree  as  their  highest  degree  obtained,  and  one  student  reported  another 
type  of  degree  as  the  highest  obtained.  A  majority  of  SAS  students  had  a  bachelor's  degree 
(64.3%),  with  5  of  the  14  students  having  a  master's  degree  (35.7%). 
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Table  1 


Demographic  Information- Rank  and  Education 


Test  Site 

Variable 

Huntsville 
(11  =  31) 

Air  Force  Institute 
of  Technology 
(AFIT) 

(11=15) 

Systems  Acquisition 
School 
(SAS) 

Ol  =14) 

Totals 

Service 

Air  Force 

5 

13 

14 

32 

Army 

17 

0 

0 

17 

Marine 

0 

1 

0 

1 

Navy 

4 

0 

0 

4 

FAA 

3 

0 

0 

3 

Industry 

1 

0 

0 

1 

SAF 

1 

0 

0 

1 

Other 

0 

1 

0 

1 

Grade 

01-03 

3 

4 

12 

19 

04  &  up 

11 

2 

2 

15 

GS-l-GS-10 

0 

1 

0 

1 

GS-11  &  up 

16 

8 

0 

24 

N/A 

1 

0 

0 

1 

Degree 

None 

NA 

0 

0 

0 

Bachelor 

NA 

7 

9 

16 

Master 

NA 

7 

5 

12 

Other 

NA 

1 

0 

1 

In  terms  of  years  of  acquisition  experience,  AFIT  students  had  fewer  years  (with  a  mean 
of  3.37  years  or  M  =  3.37)  than  SAS  students  (M  =  6.41),  but  the  years  spanned  a  wider  range 
(with  a  standard  deviation  of  6.03  or  SD  =  6.03)  (Table  2).  A  majority  of  the  AFIT  students  had 
no  acquisition  experience  (60%),  although  the  range  extended  to  one  student  having  18  years' 
experience.  Acquisition  experience  for  SAS  students  was  distributed  a  bit  more  evenly,  with  the 
highest  number  of  years  being  14. 
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Table  2 


Demographic  Information-- Work  Experience 


Test  Site 

Variable 

Huntsville 

(n=3i) 

AFIT 
(n=  15) 

SAS 

(11=14) 

Totals 

PMC  Experience 

None 

0 

15 

14 

29 

PMC  1 

31 

0 

0 

31 

PMC  1  &  2 

0 

0 

0 

0 

Years  Acquisition 
Experience 

M 

NA 

3.37 

6.41 

sn 

NA 

6.03' 

4.17 

Years  Program 

Office  Experience 

M 

NA 

1.33 

3.52 

SD 

NA 

2.25 

3.63 

Years  Military  or 
Federal  Service 

M 

NA 

11.47 

7.95 

sn 

NA 

6.65 

5.31 

Note.  NA  =  Data  not  available. 

A  similar  profile  was  found  for  the  experimental  sites  with  regard  to  the  number  of  years 
students  recorded  for  program  office  experience,  with  SAS  students  having  a  higher  mean 
number  of  years'  experience  (M  =  3.52).  The  majority  of  AFIT  students  (nine)  did  not  have  any 
program  office  experience  (60%).  However,  AHT  students  had  more  years  of  military/federal 
service  (M  =  1 1.47)  than  had  SAS  students  (M  =  7.95). 
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Training  Conditions 

Control  Group  Conditions 

The  students  in  the  control  group  (Huntsville,  AL)  were  DSMC  students  taking  the  6- 
week  PMC(I)  course.  DSMC  assumed  that  these  students  spent  between  15  and  75  minutes  of 
preparation  time  for  each  hour  of  residential  classroom  instruction.  The  classroom  presentation 
of  the  TPM  material  (the  same  material  that  was  taught  in  the  transported  lessons)  was 
interspersed  over  3  of  the  6  weeks  of  the  course.  Most  of  the  instruction  was  given  in  a 
lecture/discussion  style  to  a  class  of  31  students.  No  homework  was  given  other  than  the 
preparation  readings. 

Experimental  Group  Conditions 

No  additional  time  was  required  of  the  students  beyond  that  used  in  studying  the 
transportable  lesson  materials.  Students  worked  individually  and  at  their  own  pace.  Some 
students  did  the  lessons  at  their  own  work  sites  and  others  did  them  at  computer  work  stations 
close  to  their  work  sites.  Because  of  this  flexibility,  they  were  able  to  coordinate  the  lesson 
sessions  with  their  regular  work  schedules. 

Final  Testing  Conditions 

A  final  test  was  given  to  all  students  in  both  the  control  and  experimental  groups.  The 
test  items  were  developed  by  both  DSMC  subject  matter  experts  and  by  instructional  designers  at 
NAVPERSRANDCEN.  The  55  multiple-choice  items  were  based  on  lesson  objectives.  Longer 
lessons  with  more  objectives  required  more  questions  than  shorter  lessons. 

The  students  who  received  the  residential  classroom  instruction  were  tested  in  the  fifth 
week  of  the  6-week  course  after  the  seven  TPM  lessons  had  been  presented.  The  TPM  lessons 
were  interspersed  with  lessons  in  other  topic  areas  throughout  the  5  weeks.  Students  who 
received  the  transportable  materials  were  given  only  the  TPM  lessons.  They  were  tested 
individually  after  they  finished  the  seven  lessons.  These  students  did  not  receive  any  other 
training  modules. 


RESULTS 

All  analyses  were  conducted  using  the  BMDP  statistical  software  package  (Dixon,  1988). 
Means  and  standard  deviations  for  the  total  number  correct  on  the  final  test  for  the  control 
(Huntsville)  and  experimental  groups  (AFIT  and  SAS)  are  presented  in  Table  3.  A  one-way 
analysis  of  variance  was  conducted  on  the  total  number  correct  to  test  for  differences  between 
the  means  of  all  three  groups. 
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Table  3 


Means  and  Standard  Deviations  for 
Final  Test  Scores  by  Site 
(Number  Correct  Out  of  55  Items) 


Test  Site 

Measure 

Huntsville 

AFIT 

SAS 

(11  =  31) 

(A  =15) 

II 

3 

M 

33.19 

40.80 

39.64 

sn 

6.43 

6.33 

6.36 

Results  of  the  analysis  of  variance  yielded  significant  differences  (E(2,57)  =  9.24,  £<.01 ). 
Although  sample  sizes  for  the  groups  differed,  the  Levene  test  for  equal  variances  indicated  no 
significant  difference  between  variances.  The  Scheffe  pairwise  comparison  test  indicated  that 
the  mean  total  score  for  AFIT  (M  =  40.80)  was  significantly  higher  than  the  mean  total  score  for 
Huntsville  (M  =  33.19)  at  the  p<.01  level.  The  Scheffe  test  also  indicated  a  higher  mean  score 
for  SAS  (M  =  39.64)  over  the  mean  score  for  Huntsville  at  the  p<.05  level.  The  Scheffe  test 
yielded  no  significant  differences  between  AFIT  and  SAS,  indicating  no  performance  differences 
on  the  final  test  between  the  two  experimental  sites. 

The  mean  final  test  scores  and  standard  deviations  for  the  residential,  transportable  print, 
and  transportable  computer  courses  are  presented  in  Table  4.  Results  of  the  analysis  of  variance 
comparing  the  mean  final  total  scores  were  significant  CE(2,57)  =  9.83,  £<.01 ).  The  Scheffe  test 
indicated  that  the  mean  total  score  for  the  transportable  print  version  of  the  course  (M  =  38.85) 
was  significantly  higher  than  mean  final  score  for  the  residential  course  (M  =  33.19)  at  the  p<.05 
level.  The  mean  final  score  for  the  transportable  computer  course  (M  =  41.38)  was  also 
significantly  higher  than  that  of  the  residential  course  at  the  p<.01  level.  No  significant 
differences  were  found  between  the  print  and  computer  versions  of  the  course. 
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Table  4 


Means  and  Standard  Deviations  for 
Final  Test  Scores  by  Course 
(Number  Correct  Out  of  55  Items) 


Course 

Measure 

Residential 

Print 

Computer 

(U  =31) 

(11=13) 

(11=16) 

M 

33.19 

38.85 

41.38 

sn 

6.43 

6.50 

6.01 

The  means,  standard  deviations,  and  ranges  of  times  for  completion  of  each  lesson  were 
recorded  for  the  experimental  groups  only.  Results  are  presented  in  Table  5.  Times  to  complete 
the  lessons  varied  considerably  from  lesson  to  lesson  and  from  student  to  student.  Coordinators 
reported  that  few  students  had  questions  about  the  content  of  the  materials.  Separate  i-tests 
assessing  differences  between  the  two  experimental  sites  indicated  only  one  statistically 
significant  result.  Students  at  AFIT  took  significantly  longer  (87  minutes)  than  SAS  students 
(54.50  minutes)  to  complete  TPM  3. 

I-tests  assessing  differences  between  print  and  computer  versions  on  time  to  complete 
lessons  yielded  no  significant  differences. 

Separate  l-tests  were  also  performed  on  the  attitude  assessment  items  for  the 
experimental  groups  to  measure  differences  between  their  responses.  The  students  ranked  four 
10-scale  items  (see  Appendix  B,  Overall  Lesson  Feedback  Form).  The  first  item  addressed  the 
issue  of  "beneficial"  learning  versus  "waste  of  time";  the  second  item  concerned  the  relative 
difficulty  or  ease  with  which  the  student  was  able  to  follow  the  lessons;  the  third  item  concerned 
the  degree  to  which  the  student  found  the  material  to  be  motivating;  the  fourth  item  addressed 
perceived  mastery  of  the  content.  The  means  and  standard  deviations  for  the  four  items  for  both 
experimental  groups  were  calculated.  Those  tests  that  resulted  in  significant  differences  are 
presented  in  Table  6.  If  the  Levene  test  for  differences  between  variances  was  significant,  a 
separate  variance  estimate  was  used.  If  the  Levene  test  did  not  indicate  significant  differences, 
the  pooled  variance  estimate  was  used. 
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Table  5 


Means  and  Standard  Deviations  for  Time  Required  by 
Experimental  Groups  (AFIT  and  SAS)  to  Complete  TPM  Lessons 

(Minutes) 


Experimental  Group 

TPM  Lesson  Number3 

AFIT 

SAS 

Combined 

1.  TPM  1 

M 

113.33 

96.43 

105.17 

sn 

88.32 

24.72 

62.25 

2.  TPM  2 

M 

119.07 

112.71 

116.00 

sn 

58.20 

33.06 

47.03 

3.  TPM  3 

M 

87.00 

54.50 

71.31 

sn 

47.50 

21.00 

40.08 

4.  TPM  4 

M 

65.00 

41.15 

53.93 

sn 

46.48 

25.33 

39.40 

5.  TPM  5/6 

M 

190.13 

148.71 

170.14 

sn 

111.11 

50.12 

88.22 

6.  TPM  7 

M 

151.33 

66.79 

110.52 

sn 

166.09 

37.85 

127.70 

7.  TPM  8 

M 

106.36 

72.67 

90.81 

sn 

76.23 

44.75 

64.78 

aEarly  in  the  design  of  the  lessons,  developers  created  eight  lessons,  but  later  merged  the  content  of  Lessons  5  and 
6.  Lessons  5  and  6  subsequently  became  known  as  TPM  5/6,  with  TPM  7  and  8  retaining  their  original  numbers. 
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Table  6 


Means  and  Standard  Deviations  for  Attitude  Assessment  Items 
(Experimental  Group  Only) 


Measure 

AFIT 

SAS 

t 

df 

Beneficial  vs.  Waste  of  Time 

TPM  7  M 

3.93 

6.64 

-2.96* 

27.0 

SD 

2.34 

2.59 

Motivating  vs.  Just  More  Work 

TPM  8  M 

3.86 

6.00 

-2.55* 

24.0 

SD 

1.88 

2.41 

Mastery  vs.  Uncertainty 

TPM  1  M 

1.73 

3.93 

-2.61* 

14.3 

SD 

.70 

3.08 

TPM  2  M 

2.20 

5.07 

-3.30** 

15.6 

sn 

1.01 

3.10 

TPM  8  M 

2.93 

5.08 

-2.23* 

14.7 

sn 

1.38 

3.09 

Notes:  *p<.05. 

**p<.01. 

A  significant  difference  was  found  for  the  student's  perception  of  mastery  on  TPM  1, 
TPM  2,  and  TPM  8  between  the  two  experimental  groups,  indicating  the  AFIT  students  felt  that 
they  mastered  the  content  of  the  lessons  better  than  did  SAS  students. 
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AFIT  students  found  TPM  7  more  beneficial  (M  =  3.93)  than  did  SAS  students  (M  = 
6.64);  no  other  significant  differences  existed  between  the  groups  with  regard  to  perceived 
benefits  derived  from  the  other  lessons.  AFIT  students  (M  =  3.86)  found  TPM  8  to  be 
significantly  more  motivating  than  did  the  SAS  students  (M  =  6.00). 

No  significant  differences  were  found  between  AFIT  and  SAS  regarding  the  degree  of 
difficulty  involved  in  following  the  lessons. 

No  significant  differences  were  found  when  the  four  attitude  items  were  analyzed  by 
course  for  the  two  experimental  groups.  Print  versions  and  computer  versions  did  not  yield  any 
significant  differences  across  lessons  for  any  of  the  attitude  assessment  questions. 

Through  written  comments,  we  learned  that  the  coordinator  at  one  experimental  site 
initially  had  some  difficulty  persuading  people  to  use  the  computer  version.  People  seemed  to 
lack  confidence  with  computers  or  computer  techniques.  Yet,  by  the  end  of  the  evaluation 
period,  all  of  the  computer  volunteers  were  positive  about  using  the  computerized  lessons  and 
felt  that  they  had  learned  something.  This  group  suggested  providing  a  print  copy  of  the 
materials  to  supplement  the  computer  version  to  reinforce  what  they  learned  on  line  and  to  keep 
for  future  reference. 

I-tests  conducted  to  assess  the  differences  between  mean  years  of  acquisition  experience, 
program  office  experience,  and  military/federal  service  indicated  no  significant  differences 
between  subjects  at  AFIT  and  SAS. 


DISCUSSION 


The  transportable  courseware  yielded  higher  scores  on  the  final  test  than  did  the 
residential  courseware,  supporting  the  hypothesis  that  the  transportable  courseware  would  at 
least  yield  equivalent  results  on  an  objective  test. 

There  were  no  significant  test  score  differences  between  students  who  received  the  print 
version  and  those  who  received  the  computer  version,  supporting  the  hypothesis  that  the 
computer  version  would  produce  the  same  results  as  the  print  version.  Also,  since  no  differences 
were  found  between  performance  scores  from  the  two  experimental  sites,  it  is  possible  that 
having  computerized  transportable  materials  available  at  a  shared  computer  near  the  work  site  is 
just  as  effective  as  having  them  available  at  individual  work  sites. 

There  may  be  several  reasons  why  the  transportable  courses  yielded  higher  test  results. 
The  classroom  students  were  tested  after  receiving  instruction  not  only  on  the  seven  TPM 
lessons,  but  also  after  instruction  on  other  topics.  The  students  receiving  the  transportable 
courseware  completed  only  the  TPM  lessons,  then  received  the  TPM  final  test.  Interspersing  the 
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TPM  lessons  with  other  types  of  lessons  may  have  reduced  the  residential  group’s  performance 
on  the  final  TPM  test  because  of  additional  information  that  had  to  be  considered  and  learned. 
Conversely,  it  could  be  argued  that  the  additional  lessons  have  the  potential  to  reinforce  the  TPM 
lesson  material,  since  the  TPM  lessons  contain  the  tools  (such  as  basic  math  operations)  that 
program  managers  use  throughout  other  phases  of  program  management  and  apply  to  other 
lessons. 

Another  reason  why  the  transportable  lessons  resulted  in  higher  final  test  scores  may  be 
that  classroom  instructors  used  different  instructional  methods  than  those  used  in  transportable 
materials.  Clark  (1985)  suggests  that  many  studies  that  compare  media  do  not  distinguish  the 
medium  from  the  instructional  method.  He  reports  that  characteristics  typically  found  in 
computer-based  instruction  are  not  generally  found  in  classroom  lectures,  such  as  examples 
matched  with  non-examples  that  illustrate  the  lesson  objective,  individualized  pacing,  a 
correspondence  between  instruction  and  test  items,  and  corrective  feedback  after  responses. 

The  transportable  lessons  for  this  project  were  designed  and  developed  using  a  systematic 
method  (the  SEID  process).  The  lessons  were  written  from  a  set  of  lesson  specifications  and 
lesson  objectives,  and  the  test  items  on  the  final  TPM  test  reflected  those  objectives.  At 
Huntsville,  the  five  instructors  may  have  focused  on  issues  other  than  those  represented  by  the 
lesson  objectives.  The  instructors  may  or  may  not  have  covered  material  that  appeared  on  the 
final  test.  Because  the  teaching  methods  used  by  the  instructors  may  have  differed  greatly  within 
the  control  group  setting  and  because  control  group  learning  of  the  TPM  module  took  place 
within  a  larger  context  (i.e.,  as  part  of  the  PMC  introduction),  these  score  differences  cannot  be 
attributed  to  the  transportable  media  alone. 

In  addition,  the  self-paced  nature  of  the  transportable  materials  could  have  affected  the 
test  scores.  Students  in  the  experimental  groups  could  complete  all  or  parts  of  a  section  of  a 
lesson,  could  review  materials  at  any  time,  or  perform  the  practice  and  test  items  more  than  once. 
Students  attending  lectures  at  the  residential  school  might  have  found  it  more  difficult  to  review 
materials.  For  example,  if  questions  could  not  be  answered  during  the  scheduled  lecture,  it  may 
have  been  difficult  to  talk  with  the  instructor  outside  of  class  time.  Also,  the  self-paced  students 
might  have  chosen  to  study  the  materials  when  they  were  most  alert  and  best  able  to  attend  to 
new  information. 

The  transportable  lessons  were  designed  to  be  interactive.  All  students  were  expected  to 
respond  to  a  variety  of  scenarios  and  answer  many  questions  during  the  lessons.  In  typical 
classrooms,  often  only  a  few  students  participate  in  discussions.  Perhaps  the  transportable 
lessons  stimulated  more  active  learning  by  all  the  students. 
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CONCLUSIONS  AND  RECOMMENDATIONS 


The  goal  of  this  project  was  to  design,  develop,  test,  and  evaluate  a  set  of  transportable 
training  materials  that  introduce  program  management  concepts  to  acquisition  managers.  It  was 
hypothesized  that  these  training  materials  would  result  in  performance  gains  that  were  at  least 
equal  to  those  obtained  by  students  who  received  residential  instruction.  The  goal  was  achieved 
and  the  hypothesis  was  confirmed.  In  fact,  students  who  used  transportable  courseware 
performed  better  on  the  final  test  than  did  the  residential  students. 

We  did  not  hypothesize  that  the  transportable  lessons  would  result  in  instruction  superior 
to  conventional  classroom  instruction.  The  purpose  of  this  investigation  was  to  determine  if  the 
transportable  courseware  could  serve  as  an  adequate  substitute  for  classroom  instruction  under 
the  premises  that  transportable  instruction  is  more  economical  to  produce  and  can  reach  far 
greater  numbers  of  students.  Transportable  instruction  was  viewed  as  a  vehicle  for  alleviating 
high  student  demand,  inadequate  facilities  for  meeting  this  demand,  and  the  travel  costs 
associated  with  attending  residential  classroom  instruction. 

When  developing  future  courses,  it  remains  important  to  determine  which  skills  are  best 
taught  via  transportable  lessons  using  the  GTET  guidelines.  It  is  probably  inappropriate  to 
develop  transportable  courseware  for  every  lesson  in  a  residential  program.  Some  skills  may  be 
more  readily  taught  within  a  classroom  setting  and  may  be  more  amenable  to  group  discussion 
and  exchange. 

Because  use  of  the  print  and  computer  versions  of  the  lessons  did  not  result  in  any 
differences  in  student  performance,  it  may  be  more  efficient  to  develop  only  print  versions  if 
economy  is  important.  We  suggest  this  because:  (1)  print  materials  are  generally  less  costly  and 
less  time-consuming  to  develop  and  produce  than  computer  media;  (2)  print  materials  are  more 
portable;  (3)  there  are  no  hardware  requirements;  and  (4)  recent  analysis  indicates  that  a  specific 
medium  will  not  provide  learning  benefits  unattainable  from  other  media  (Clark  &  Sugrue,  in 
press;  Levie  &  Dickie,  1973;  Schramm,  1977). 

Recent  research  supports  the  idea  that  while  new  electronic  media  (such  as  computers 
and  videodiscs)  may  not  provide  psychological  advantages  in  learning,  they  may  provide 
economic  advantages  when  compared  with  residential  instruction  in  terms  of  expended  resources 
and  speed  of  learning  (Clark  &  Sugrue,  in  press).  These  findings  are  consistent  with  this  effort's 
purpose  of  developing  the  transportable  lessons  to  help  ease  the  costs  of  attending  residential 
courses  and  to  allow  students  to  complete  courses  at  their  own  pace. 
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APPENDIX  A 

TPM  LESSON  OVERVIEWS 


A-0 


TOOLS  FOR  PROGRAM  MANAGEMENT  (TPM) 
LESSON  OVERVIEW 


The  introductory  component  of  the  Defense  Systems  Management  College's  Program 
Management  Course  (PMC  (I))  is  designed  to  cover  basic  topics  in  defense  acquisition.  It 
contains  seven  modules  of  instruction.  The  Tools  for  Program  Management  (TPM)  lesson  fits 
into  PMC  (I)  as  highlighted  below: 

PMC  (I)  MODULES 

1.  Defense  Acquisition  Environment  and  Process  (DAEP) 

2.  Tools  for  Program  Management  (TPM) 

3.  Program  Planning  and  Control  (PPC) 

4.  Resource  Estimation  and  Management  (REM) 

5.  Govemment/Contractor  Relationship  (GCR) 

6.  System/Product  Definition  (SPD) 

7.  Executive  Skills  Development  (ESD) 

The  TPM  module  consists  of  seven  lessons:  * 


TPM  LESSONS 

TPM  01 

Mathematics  for  Resource  Estimation 

TPM  02 

Economic  Analysis 

TPM  03 

Work  Planning  and  Definition 

TPM  04 

System  Effectiveness 

TPM  05/06 

Quantitative  Decision  Making 

TPM  07 

Non-Quantitative  Decision  Making 

TPM  08 

Trade-off  Analysis 

'Early  in  the  design  of  the  lessons,  developers  created  eight  lessons,  but  later  merged  the  content  of  Lessons  5  and 
6.  Lessons  5  and  6  subsequently  became  known  as  TPM  5/6,  with  TPM  7  and  8  retaining  their  original  numbers. 
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TPM  01 

MATHEMATICS  FOR  RESOURCE  ESTIMATION 


Prerequisites 

None 

Lesson  Introduction 

This  lesson  covers  the  mathematics  you  will  need  to  understand  learning 
curveanalysis,  which  is  taught  in  FM  010  (the  first  lesson  of  the  Funds 
Managementmodule).  Each  segment  will  cover  skills  relevant  to  learning 
curveanalysis,  in  which  you  predict  production  costs  for  future  lots  based  on 
datafrom  previous  lots. 

Lesson  Objectives 

At  the  end  of  this  lesson  you  will  be  able  to: 

1.  Plot  data  points  on  coordinate  graph  paper  and  log-log  paper. 

2.  Forecast  the  value  of  y  given  a  value  of  x  and  data  that  is  already  plotted. 

3.  Solve  for  y  in  an  equation  when  given  a  value  for  x. 

4.  Solve  for  x  and  y  in  simultaneous  equations. 

5.  Use  a  calculator  with  a  yx  key  to  solve  equations  with  either  positive  or 
negative  exponents. 

6.  Use  logarithms  to  solve  equations. 

7.  Explain  the  components  of  the  mathematical  equation  for  a  straight  line: 
y  =  mx  +  b  and  the  learning  curve:  y  =  axb. 

8.  Determine  the  equation  of  a  line,  given  data  points  (x,y)  or  a  plotted  linear 
graph. 

9.  Solve  for  y,  given  data  points  (x,y)  or  a  learning  curve  plotted  on  a  graph  and 
values  of  x. 

10.  Describe  the  use  of  the  visual  method  of  regression  analysis  to  determine 

whether  your  data  is  accurate,  sufficient  and  linear,  or  curvilinear. 

11.  Describe  the  use  of  the  least  squares  method  of  linear  regression  analysis. 

12.  Describe  how  to  conduct  regression  analysis  on  data  that  is  curvilinear. 

13.  Use  the  three  measures  of  goodness  of  fit  (standard  error  of  the  estimate 

(SEE),  correlation  coefficient,  and  coefficient  of  determination)  to  determine 
how  well  the  independent  variable  predicts  the  dependent  variable. 

Required  Resources 

You  will  need  a  calculator  with  a  yx  key  for  this  lesson. 
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TPM  02 

ECONOMIC  ANALYSIS 


Prerequisites 

None 

Lesson  Introduction 

As  a  program  manager,  you  should  be  aware  of  how  companies  that  you  work 
with  operate- -their  motivation,  their  risks,  and  how  they  make  investment 
decisions.  In  this  lesson,  economic  analysis,  you  will  learn  about  basic  economic 
principles  and  how  to  apply  analysis  techniques  to  make  investment  decisions 
yourself. 

Lesson  Objectives 

At  the  end  of  this  lesson  you  will  be  able  to: 

1.  Explain  what  is  meant  by  the  time  value  of  money. 

2.  Determine  the  investment  that  yields  the  greatest  future  value. 

3.  Determine  the  best  investment  opportunity  by  discounting  future  value  to 
present  value. 

4.  Explain  the  relationship  between  time  value  of  money,  interest  rate,  and 
investor  risk. 

5.  Explain  the  seven  risk  factors  associated  with  interest  rates. 

6.  Define  economic  terms  relating  to  the  use  of  capital. 

7.  Define  economic  analysis. 

8.  Explain  reasons  for  using  Economic  Analysis  in  DoD  systems  acquisition. 

9.  Describe  economic  profit  and  return  on  investment. 

10.  Explain  the  role  of  cost  of  capital  in  evaluating  investment  techniques. 

11.  Calculate  the  payback  period  to  determine  the  best  project  to  invest  in. 

12.  Calculate  the  Net  Present  Value. 

13.  Calculate  the  Internal  Rate  of  Return. 


Required  Resources 

You  will  need  a  calculator  with  a  yx  key  for  this  lesson. 
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TPM  03 

WORK  PLANNING  AND  DEFINITION 


Prerequisites 

None 

Lesson  Introduction 

One  of  the  most  useful  tools  for  program/project  managers,  in  both  the 
Department  of  Defense  (DoD)  and  industry,  is  the  work  breakdown  structure 
(WBS).  The  WBS,  in  its  several  forms,  is  extremely  valuable  as  managers 
engage  in  planning,  defining,  and  controlling  their  work. 

A  WBS,  if  well  written,  defines  the  program/project's  total  objectives;  it  relates 
the  various  work  efforts  (parts)  to  the  overall  product  or  end  objective  (whole) 
and  provides  a  framework  for  the  required  budgeting  and  managing  of  resources. 

This  lesson  presents  an  overview  of  work  breakdown  structures.  Many  terms  are 
introduced  in  this  lesson.  The  terms  are  defined  throughout  the  lesson  and  are  in 
accordance  with  MIL-STD-881A,  Work  Breakdown  Structures  for  Defense 
Materiel  Items. 


Lesson  Objectives 

At  the  end  of  this  lesson  you  will  be  able  to: 

1.  Explain  the  purpose  and  importance  of  the  WBS. 

2.  Define  the  following  terms: 

Tree  diagram 
WBS  elements 
WBS  levels 
Numbering  system 
Work  package 
WBS  dictionary 

3.  Explain  the  four  types  of  WBSs,  including  the  purpose  and  the  timing  of  each 
in  the  acquisition  process. 

4.  Use  MIL-STD-881A  to  prepare  a  Project  Summary  WBS,  including 
numbering,  given  a  list  of  15-20  WBS  elements. 

Required  Resources 

You  will  need  a  copy  of  MIL-STD-881 A  for  this  lesson. 
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TPM  04 

SYSTEM  EFFECTIVENESS 


Prerequisites 

None 

Lesson  Introduction 

This  lesson  presents  an  overview  of  system  effectiveness.  Every  system  is 
designed  to  respond  to  a  stated  need  or  requirement. 

System  effectiveness  may  be  defined  as  the  measure  of  the  extent  to  which  a 
system  may  be  expected  to  achieve  a  set  of  specific  mission  requirements. 

System  effectiveness  may  be  thought  of  as  a  function  of  availability, 
dependability,  capability,  and  other  variables.  Our  discussion  will  define  those 
terms  necessary  to  understand  the  concept  of  system  effectiveness.  The 
definitions  we  provide  in  this  lesson  are  appropriate  for  either  civilian  or  military 
System  Effectiveness  discussions. 

Lesson  Objectives 

At  the  end  of  this  lesson  you  will  be  able  to: 

1.  Define  the  following  terms:  availability,  dependability,  capability. 

2.  Explain  the  relationship  of  the  four  types  of  models  used  to  evaluate  system 
effectiveness. 

-Time  line  model 
-Deterministic  model 
-Probabilistic  model 
-Relational  (Interface)  model 

3.  Explain  when  utility  theory  should  be  applied  to  determine  system 
effectiveness. 

Required  Resources 
None 
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TPM  05/06 

QUANTITATIVE  DECISION  MAKING 


Prerequisites 

None 

Lesson  Introduction 

Decision  making  is  probably  one  of  the  most  difficult  and  complex  tasks  program 
managers  face.  It  is  also  one  of  the  most  important  aspects  of  the  job.  Three 
lessons  in  the  TPM  series  present  the  following  decision-making  techniques  to 
help  you  in  your  role  as  decision  maker:  quantitative  decision  making,  non- 
quantitative  decision  making,  and  trade-off  analyses. 

This  lesson  presents  a  quantitative  decision-making  technique  called  decision  tree 
analysis  (or  expected  value  analysis)  which  is  based  on  the  application  of 
expected  value.  Quantitative  decision  making  involves  the  estimation  and  use  of 
probabilities  to  draw  inferences  about  potential  outcomes. 

Lesson  Objectives 

At  the  end  of  this  lesson  you  will  be  able  to: 

Segment  1 

1.  Use  your  knowledge  of  probability  to  predict  the  chance  of  an  event  occurring  (when 
outcomes  are  equally  probable). 

2.  Identify  the  characteristics  of  the  different  types  of  distributions  used  to  model  data. 

3.  Determine  the  measures  of  central  tendency  and  measures  of  dispersion  of  a 
probability  distribution. 

4.  Predict  or  estimate  the  chance  of  an  event  occurring  if  you  know  the  mean  and 
standard  deviation  of  a  distribution  (and  can  assume  a  normal  distribution). 

Segment  2 

1.  Identify  when  it  is  appropriate  to  use  decision  tree/expected  value  analysis. 

2.  Describe  the  following  components  of  a  decision  tree: 

Nodes :  act  node,  event  node 

Elements:  action  (choice),  event,  probability,  outcome 

3.  Calculate  the  expected  value  of  each  action. 
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Segment  3 

1.  Make  a  decision  based  on  expected  value  analysis,  when  given  a  decision  tree. 


Segment  4 

1.  Explain  how  the  following  techniques  relate  to  decision  tree  analysis:  sensitivity 
analysis,  uncertainty  analysis,  risk  analysis. 

2.  Explain  how  the  point  of  indifference  can  be  used  in  decision  tree  analysis  to 
determine  the  sensitivity  of  a  decision. 

3.  Use  a  decision  tree  to  calculate  the  point  of  indifference. 

4.  Evaluate  your  decision  using  the  point  of  indifference. 

Required  Resources 

You  will  need  a  calculator  for  this  lesson. 
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TPM  07 

NON-QUANTITATIVE  DECISION  MAKING 


Prerequisites 

None 

Lesson  Introduction 

There  are  several  well-known  formal  approaches  to  non-quantitative  decision 
making,  including  the  Kepner-Tregoe  (K-T)  approach  and  the  creative  problem 
solving  (CPS)  approach.  This  lesson  introduces  the  Kepner-Tregoe  approach  to 
illustrate  the  advantages  of  a  formal  non-quantitative  approach. 

Even  a  one-week  workshop  on  the  K-T  or  CPS  approach  is  only  a  starting  point, 
so  this  lesson  is  not  intended  to  teach  you  to  be  proficient  in  applying  the  K-T 
techniques.  Instead,  the  purpose  is  to  increase  your  awareness  of  your  own 
decision-making  process  and  to  present  ideas  that  you  may  be  able  to  incorporate 
into  your  own  process. 

Lesson  Objectives 

At  the  end  of  this  lesson  you  will  be  able  to: 

1.  Identify  the  four  basic  processes  of  the  Kepner-Tregoe  approach. 

2.  Explain  the  stages  used  in  the  situation  appraisal  process. 

3.  Explain  the  main  steps  in  problem  analysis  and  apply  the  steps  to  a  case  study. 

4.  Explain  the  main  steps  in  decision  analysis. 

5.  Explain  the  main  steps  in  developing  an  implementation  plan  using  potential 
problem  analysis. 

Required  Resources 
None 
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TPM  08 

TRADE-OFF  ANALYSIS 


Prerequisites 

It  is  recommended  that  you  complete  TIJM  01,  Mathematics  for  Resource 
Estimation,  and  TPM  05/06,  Quantitative  Decision  Making,  before  starting  this 
lesson. 


Lesson  Introduction 

It  is  appropriate  to  use  a  formal  decision  analysis  method  when:  (1)  a  problem  is 
complex,  (2)  a  problem  is  important,  and/or  (3)  you  must  document  and 
communicate  your  rationale  for  a  course  of  action  to  a  fmal  decision  maker. 

The  formal  methods  for  decision  making  and  problem  solving  are  either 
quantitative  or  non-quantitative.  Trade-off  analysis  is  a  quantitative  method,  as  is 
expected  value  analysis  (discussed  in  TPM  05.)  The  Kepner-Tregoe  method 
(discussed  in  TPM  07)  is  one  of  many  non-quantitative  methods. 

This  lesson  presents  an  overview  of  a  generic  trade-off  analysis  method,  which  is 
a  formal  decision  making  method  for  evaluating  a  set  of  alternative  concepts  or 
designs.  It  is  beyond  the  scope  of  this  lesson  to  teach  all  the  details  and  levels  of 
complexity  of  the  method. 

Lesson  Objectives 

At  the  end  of  this  lesson  you  will  be  able  to: 


1.  Identify  when  to  use  the  expected  value  vs.  the  trade-off  analysis  method. 

2.  Explain  the  six  steps  for  conducting  a  trade-off  analysis. 

3.  Perform  Step  5,  Evaluating  Alternatives,  of  trade-off  analysis. 

4.  Perform  Step  6,  Conducting  Sensitivity  Analysis,  of  trade-off  analysis. 


Required  Resources 

You  will  need  a  calculator  for  this  lesson. 
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APPENDIX  B 

LESSON  FEEDBACK  AND  DEMOGRAPHIC  SURVEY  ITEMS 
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DEFENSE  SYSTEMS  MANAGEMENT  COLLEGE 
Program  Management  Course,  Part  I  (PMC  (I)) 


FEEDBACK  FORM  PACKET 

We  are  exploring  the  conversion  of  parts  of  the  PMC(I)  Course  to  a  self-study  option.  We  are 
evaluating  the  first  lessons  converted  to  this  self-study  format.  We  are  interested  not  only  in 
testing  your  knowledge  of  the  subject  matter,  but  also  how  well  the  materials  teach.  The 
feedback  you  provide  on  these  lessons  will  help  us  improve  the  instruction  for  these  and  other 
lessons  in  the  acquisition  field. 


1.  Name  or  ID  Number  (ask  your  evaluation  coordinator): 


2.  Job  title: _ 

3.  Organization:. 


Work  address:. 


4.  Work  phone: _ 

5.  Job  supervisor  name: _ 

6.  Are  you  a  PMC  student? _ Yes _ No 

7.  Have  you  been  a  PMC  student  in  the  past? _ Yes _ No 

If  YES,  what  class  were  you  in? _ 


Feedback  Form  Packet 
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Program  Management  Courses 


8.  Please  complete  the  following  background  information. 

Military  or  Civilian  Grade _ 

Military  Organization _ 

Specialty  or  Occupational  Code _ 

What  formal  training  have  you  had  in  Systems  Acquisition? 


Total  Military  or  Federal  Service _ years 

Highest  Academic  Degree _ 

Military  Schools  (C/S) _ (SSC). 

Last  or  Current  Job  Assignment _ 

Projected  Next  Assignment _ 

Experience  in  Systems  Acquisition _ years 

Experience  in  a  Program  Office _ years 
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OVERALL  LESSON  FEEDBACK  FORM 


Lesson:  TPM  01  COMPUTER  VERSION  _  (or)  PRINT  VERSION  _ 

How  much  time  did  it  take  to  complete  this  lesson?  _ min. 

Date 


Please  circle  a  number  for  each  item  that  you  feel  best  describes 
the  lesson.  Then  write  comments  or  suggestions  for  each  item. 

1 .  Beneficial  Waste  of  time 

1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 - 10 

Comments : 


2 .  Difficult  to  follow  Easy  to  follow 

1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 - 10 

Comments : 


3.  Motivating  Just  more  work 

1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 - 10 

Comments : 


4.  Feel  I've  mastered  the  content  Not  sure  what  it  taught 

1 - 2 - 3 - 4 - 5 - 6 - 7 - 8 - 9 - 10 

Comments : 


5.  Do  you  still  have  any  unanswered  questions?  _  Yes  _  No 

If  yes,  what  are  they? 
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These  items  are  to  be  completed  AFTER  you  have  completed  all  of 
the  lessons  assigned  to  you. 


6.  Which  self-study  lesson (s)  did  you  take?  Circle  all  that 
apply. 

Print  Version 

TPM  01  02  03  04  05/06  07  08 


Computer  Version 

TPM  01  02  03  04  05/06  07  08 


Have  you  ever  taken  Mathematics  for  Resource  Estimation  (TPM  01) 

or  studied  the  content?  _  Yes  _  No 

If  YES,  please  explain  when  and  where. 


Have  you  ever  taken  Economic  Analysis  (TPM  02)  or  studied  the 

content?  _  Yes  _  No 

If  YES,  please  explain  when  and  where. 


Have  you  ever  taken  Work  Planning  and  Definition  (TPM  03)  or 

studied  the  content?  _  Yes  _ _  No 

If  YES,  please  explain  when  and  where. 


Have  you  ever  taken  System  Effectiveness  (TPM  04)  or  studied  the 

content?  _  Yes  _  No 

If  YES,  please  explain  when  and  where. 


Have  you  ever  taken  Quantitative  Decision  Making  (TPM  05/06)  or 

studied  the  content?  _  Yes  _  No 

If  YES,  please  explain  when  and  where. 


Have  you  ever  taken  Non-Quantitat ive  Decision  Making  (TPM  07)  or 

studied  the  content?  _  Yes  _  No 

If  YES,  please  explain  when  and  where. 
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Have  you  ever  taken  Trade-Off  Analysis  (TPM  08)  or  studied  the 

content?  _  Yes  _  No 

If  YES,  please  explain  when  and  where. 


OVERALL  COMMENTS  OR  SUGGESTIONS  ABOUT  ANY  ASPECT  OF  THE  LESSON (S) 
For  example/  what  would  you  like  to  see  changed  in  this  lesson? 

(Please  continue  on  other  side,  if  needed.) 
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INSTRUCTIONS  TO  EXPERIMENTAL  SITE  COORDINATORS 
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EVALUATION  COORDINATOR  INFORMATION 


The  goal  of  the  evaluation  is  to  obtain  feedback  from  participants  on  questions  such  as: 

-What  resources  are  required  to  support  the  lesson  materials? 

-How  long  does  it  take  each  participant  to  accomplish  each  lesson? 

-How  well  does  each  participant  perform  on  the  assessments? 

A  feedback  form  packet  will  be  provided  to  each  participant  to  gather:  1)  information  on  some 
key  background  questions  and  2)  their  comments  on  the  lesson  materials.  The  evaluation 
coordinator  will  complete  a  worksheet  for  each  participant  to  help  organize  the  administration 
and  management  of  the  lessons. 

As  a  reminder,  please  do  not  alter  any  of  the  training  materials.  Present  them  to  the  participants 
just  as  they  are  given  to  you 

For  the  Spring,  1989  evaluation,  it  would  be  best  to  have  at  least  10  participants  complete  each 
print/hardcopy  version  of  the  lessons  and  10  participants  complete  each  computer  lesson. 


FACILITATOR  HELP 

Each  participant  must  have  someone  who  can  serve  as  a  facilitator  to  helpwith  the  lesson 
content,  someone  they  can  call  with  questions  or  clarifications.  Facilitators  should  have 
completed  the  PMC  or  its  equivalent  or  should  have  had  at  least  five  years  of  experience  in 
acquisition  management.  It  is  assumed  that,  at  some  sites,  the  coordinator  will  meet  these 
requirements  and  also  serve  as  the  facilitator.  It  is  the  coordinator’s  responsibility  to  make  sure 
all  the  forms  in  this  packet  and  in  the  student  feedback  packets  are  completed  either  by  the 
coordinator  or 
by  the  facilitator. 

The  facilitator  would  be  the  first  person  to  whom  the  participant  would  direct  questions.  The 
students  need  to  know  how  to  contact  the  facilitator  and  exactly  when  the  facilitator  will  be 
available  for  consultation.  The  facilitator  also  needs  to  keep  track  of  the  time  that  he/she  spends 
helping  the  participants. 

If  more  information  is  needed,  the  facilitator  can  direct  (or  have  participants  do  so)  questions  to 
the  Evaluation  Coordinator.  The  Evaluation  Coordinator  can  contact  NPRDC  for  information: 
Barbara  Tarker  AV  553-7975,  Mike  Flaningam  AV  553-7964,  or  Vel  Hulton  AV  553-7958. 
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TASKS  FOR  EVALUATION  COORDINATORS 


OBTAIN  OR  PREPARE  SUFFICIENT  MATERIALS 
EACH  participant  will  need: 

-I  Evaluation  Coordinator  Worksheet  (participant  information,  to  be  filled  out  by  the 
evaluation  coordinator) 

-- 1  Participant  Feedback  Form  packet  per  lesson  (participant  feedback  from  lessons,  to 
be  filled  out  by  the  participant) 

-For  TPM01  and  TPM02,  each  participant  will  need  a  calculator  with  a  yx  function  key 
feature.  TPM07  and  TPM08  require  a  calculator. 

—For  TPM03,  a  copy  of  MIL-STD-881A  should  be  available. 

Each  HARDCOPY/PRINT  participant  will  need  1  Lesson  Booklet  per  lesson. 

Each  COMPUTER  participant  will  need: 

—1  Computer  User's  Guide  per  lesson, 

—A  computer  loaded  with  the  training  materials,  OR  a  set  of  floppy  disks  so  they  can 
load  the  materials  before  beginning, 

-A  schedule  when  this  computer  is  available  for  his/her  use, 

--I  Student  Disk  for  EACH  lesson  to  be  done  (have  the  participants  label  their  disks  with 
their  ID  numbers  and  the  lesson  numbers). 

It  would  be  helpful  to  label  each  set  of  lesson  materials  with  an  identification  number  unique  to 
each  participant  (and  let  the  participant  know  his/her  identification  number),  so  all  materials  can 
be  easily  identified. 

Since  the  lessons  need  to  be  completed  and  delivered  to  the  evaluation  coordinator  on  schedule, 
some  way  of  checking  on  participant  progress  might  be  necessary. 

COLLECTION  OF  MATERIALS 

After  the  participants  have  completed  the  lessons,  the  materials  will  be  collected  and  mailed  to 
the  GTET  office  in  San  Diego.  You  may  use  the  mailing  labels  supplied  to  you.  Materials  to 
return  include: 

-print/hardcopy  lessons, 

-all  floppy  disks, 

-feedback  form  packets  (please  verify  that  all  forms  have  been  completed) 

-tests  and  answer  sheets 
-evaluation  coordinator  worksheets 

-and  any  other  materials  on  which  participants  have  written  comments 

Please  mail  completed  materials  as  they  are  returned  to  you.  We  suggest  that  you  send  a  packet 
of  completed  materials  at  the  end  of  each  week.  THE  FINAL  DEADLINE  FOR  ALL 
MATERIALS  to  be  returned  to  the  evaluation  coordinator  is  01  AUGUST,  1989. 
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Some  things  to  check  for  computer  administration: 

-Location  of  computer.  (For  example,  are  the  computers  available  to  the  students  during 
the  hours  when  students  can  use  them?  Is  the  computer  area  quiet  and  free  of 
distractions?) 

-*Is  the  computer  set  up  and  loaded  (by  you  or  by  one  of  the  participants)? 

--Is  the  schedule  for  computer  usage  time  posted? 
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DISTRIBUTION  LIST 


Distribution: 

Assistant  for  Planning  and  Technical  Development  (OP-01  B2) 

Defense  Technical  Information  Center  (DTIC)  (2) 

Copy  to: 

Deputy  Under  Secretary  of  Defense  for  Total  Quality  Management 

Director,  Acquisition  Education,  Training  and  Career  Development  Policy,  OUSD  (A) 

Deputy  Assistant  Secretary  (RM&S)  OASD  (FM&P) 


